NOTICE:  When  goTeminent  or  other  drawings,  speci¬ 
fications  or  oth>er  data  are  used  for  any  puipiose 
other  than  in  coinnection  with  a  definitely  related 
government  procurement  operation,  the  U.  S. 
Government  thereby  incurs  no  responsibility,  nor  any 
obligation  whatsoever;  £ind  the  fact  that  the  Govern¬ 
ment  may  have  formulated,  furnished,  or  in  any  way 
supplied  the  said  drawings,  specifications,  or  other 
data  is  not  to  bte  regarded  by  implication  or  other¬ 
wise  as  in  any  manner  licensing  the  holder  or  any 
other  person  or  corporation,  or  conveying  any  rights 
or  pemission  to»  manufacture,  use  or  sell  any 
patented  invention  that  may  in  any  way  be  related 
thereto. 


I 


I 


CO 

CO 

morandum 

l:'^M-3173-PR 

OiUNE  1962 

•*  .  ■ 

'  '•  •ftt  * 

hoc  ? 

r 

^3*#w**#  * 

03 

C3  O 

Cni> 


DYNAMIC  PROGRAMMING, 
INTELLIGENT  MACHINES, 
AND  SELF-ORGANIZING  SYSTEMS 


Richard  Bellman 


PREPARED  FOR; 

UNITED  STATES  AIR  FORCE  PROJECT  RAND 


ks  n 


.iiJ  Lljv'..''‘ '-’-U  ^  -Liii- 


IMi 

'll': 


74e 


(l<x>ifrcn4Xtiott 


SANTA  MONICA  •  CALIFORNIA - 


I 


t  MEMORANDUM 

I  RM-3173-PR 

[  JUNE  1962 


DYNAMIC  PROGRAMMING, 
INTELLIGENT  MACHINES, 
AND  SELF- ORGANIZING  SYSTEMS 

Richard  Bellman 


This  research  is  s|>on.sore(l  by  the  United  Slates  Air  Force  under  Project  HAND  — Con¬ 
tract  No.  AF  49(6.38 ) -700  —  monitored  hy  the  Directorate  of  Development  Planning, 
Deputy  Chief  of  Staff,  Hesearch  and  Technology,  Hq  USAF.  Viev/s  or  conclusions  con¬ 
tained  in  this  Memorandum  should  not  he  interpreted  as  representing  the  olTicial  opinion 
or  policy  of  the  United  .States  Air  Force.  Permi.ssion  to  quote  from  or  reproduce  portions 
of  this  Memorandum  must  be  obtained  from  The  PAND  Corporation. 


■nu 


^XftOKUiOK 


CAllfORND 


1700  MAIN  ST  •  SANI 


-iil- 


PREFACE 

In  this  Memorandim,  the  author  dlccusses  the 
importjance  of  a  scientific  classification  of  problems  in 
accordance  with  their  structural  features,  if  optlmvim  use 
is  to  be  made  of  intelligent  machines  for  problem 
solving. 
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SUMMARY 


In  this  Memorandijm,  the  author  discusses  some  aspects 

A  = 

of  problem  formulation  and  problem  soiutlor^  In  particular ^ 
he-emphasiz^s  the  relevance  of  these  matters  to  the 
-esEolrtlng  field  of  intelligent  machines  and  indiaates  some 
connections  with  the  theory  and  application  of  dynamic 
programming^^  Most  -l4npo^tant--of^^-14T.  -he-poln^e-out  toe 
practical  application  of  scientific  philosophy  as  a  tech¬ 
nique  to  guide  research  and  to  avoid  undue  waste  of  time, 
energy,  and  talent.  y  i  . 

J 

\ 

\ 


A 

X  \ 


/ 

K 


/ 


-vil- 


CONTENI'S 

PREFACE  . . . Ill 

SUMMARY  .  V 

Section 

1.  INTRODUCTION . 1 

2.  WHAT  DO  WE  EXPECT  FROM  A  THEORY? . 1 

3.  EXPENSIVE  COMPARTMENTALIZATION . 2 

4.  GENERATION  AND  REGENERATION  .  3 

5.  HIERARCHIES  OP  PROCESSES . 4 

6.  SYSTEMS  AND  PROCESSES  .  4 

7.  INTELLIGENCE  AND  DECISION-MAKING . 5 

8.  FIRST -iEVEL  PROCESSES . 7 

9.  DISCUSSION . 8 

10.  INSTINCT . 8 

11.  SECOND-LEVEL  PROCESSES . 9 

12.  INFINITE  HIERARCHY  OF  PROCESSES  .  9 

13.  CREATIVITY . 10 

14.  INCONSISTENT  LOGICS  AND  RANDOM  AXIOM  SYSTEMS.  .  .  11 

15.  MACHINES  THAT  PLAY  CHESS  IN  THE  NIGHT . 11 

16.  DYNAMIC  PROGRAMMING . 13 

17.  MATHEMATICAL  EXPERIMENTATION . 14 


REI'^ERENCES 


15 


DYNAMIC  PR0GRAI‘1MING,  INTELLIGENT  MACHINES  AND 
SELF-ORGANIZING  SYSTEMS 


1 .  INTRODUCTION 

As  Gertrude  Stein,  author  of  such  great  truths  as  "A 
rose  is  a  rose  is  a  rose,"  lay  on  her  deathbed,  she  was 
surrounded  by  acolytes  and  devotees  who  hoped  to  glean 
one  last  bit  of  wisdom  from  her.  One  of  them,  perhaps 
the  faithful  Alice  B.  Toklas,  leaned  near  and  asked, 
"Gertrude,  Gertrude,  vjhat  is  the  answer?"  Summoning  all 
of  her  remaining  energy,  Gertrude  Stein  opened  her  eyes 
and  feebly  responded,  "What  is  the  question?" 

One  cannot  help  feeling  that  too  much  effort  has  been 
devoted  to  obtaining  answers  without  nearly  enough  effort 
being  directed  toward  formulation  of  the  proper  question. 
Consequently,  in  what  follows,  we  would  like  to  discuss 
some  aspects  of  problem  formulation  and  problem  solution. 
In  particular,  we  shall  emphasize  the  relevance  of  these 
natters  to  the  exciting  field  of  Intelligent  machines  and 
indicate  some  connections  with  the  theory  and  application 
of  dynamic  programming.  Above  all,  we  Wcint  to  point  out 
the  practical  application  of  scientific  philosophy  as  a 
technique  to  guide  research  and  to  avoid  undue  waste  of 
time,  energy,  and  talent. 

2.  V/HAT  DO  I'/E  EXPECT  FROM  A  THEORY? 

A  mathematician,  like  any  Intellectual,  is  not  at  all 
content  with  the  mere  collection  of  files  of  facts.  He  is 
well  aware  that  the  only  defense  against  being  overwhelmed 
by  masses  of  data,  accumulating  alarmingly  at  an 
exponential  rate  each  year,  is  an  understanding  of 
structural  features,  l.e.,  "theories,"  which  enable  us  to 
organize  vast  domains  of  knowledge.  Even  this  is  not 
sufficient.  As  time  goes  on,  theories  themselves  must  be 
grouped  in  such  a  way  as  to  recognize  the  basic  identity 


of  numerous  intellectual  structures  in  different  fields. 

As  vd.th  nations,  sciences  have  been  kept  needlessly  apart 
by  distinct  languages. 

If,  then,  we  wish  to  depend  upon  theory  for  control 
and  understanding  of  the  physical  universe,  it  is  essential 
that  we  devote  some  effoi’t  to  the  Intrinsic  structure  of 
theories,  a  theory  of  theories,  as  it  were. 

In  constructing  a  theory,  what  do  v/e  demand  in  the 
way  of  problem  solving?  We  ask  the  following; 

Recognition:  The  ability  to  discern  and  categorize 

questions  which  fall  within  the  province 
of  the  theory. 

Foimulatlon:  The  capacity  for  precise,  but  not  necessarily 
equivalent,  statements  of  the  problem,  using 
the  vocabulary  of  the  theory. 

Analysis;  The  possession  of  a  set  of  systematic  tech¬ 

niques  for  obtaining  the  structure  of  the 
solution  in  terms  of  the  structure  of  the 
problem. 

Computation:  The  capability  of  furnishing  numerical 
answers  to  numerical  questions. 

To  all  of  these  fundamental  requirements,  v;e  vd.sh  to 
add  one  of  operational  significance: 

Regeneration:  The  ability  to  generate  new  questions  from 
old  answers;  this  implies  flexibility, 
versatility  and  adaptability. 

3.  EXPENSIVE  COMPARTMENTALIZATION 

All  of  these  characteristics  are  intimately  inter¬ 
twined,  as  we  have  emphasized  in  a  number  of  paper’s  [2], 

[3],  [^].  Much  of  the  difficulty  that  is  experienced  in 
applying  mathematical  methods  to  the  problems  of  the 
physical  world  derives  from  the  fact  that  there  is  seldom 
a  unified  attack  upon  a  problem.  Rather,  the  scientist — 
physical,  social  or  biological — formulates  on  the  basis  of 
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hls  wide  scientific  lcno;^rledge,  but  limited  mathematical 
skill;  the  mathematician  analyzes  on  the  base  of  broad 
mathematical  skill,  limited  scientific  backgromd,  and 
limited  understanding  of  the  abilities  and  limitations 
of  digital  computers.  The  computation  is  then  done  by 
people  skilled  in  their  specialty,  but  usually  unaware  of 
the  physical  or  mathematical  origins. 

That  anything  should  emerge  from  this  sequence  of 
poor  ImpedaJice  matching  is  remarkable.  That  the  net  effort 
is  generally  inefficient,  inept,  and  Irrelevant  is  not 
remarkable  at  all.  With  these  matters  in  mind,  it  becomes 
clear  that  the  dedication  of  time  and  effort  to  classifi¬ 
cation  of  problems  is  not  an  idle  expenditure,  but  rather 
represents  an  Important  step  in  the  direction  of 
coordinating  the  work  of  many  people  in  different  fields. 

It  is  a  particularly  significant  enterprise  for  those 
interested  in  intelligent  machines,  automata,  adaptive 
processes,  and  decision  theory. 

Let  us  make  one  further  point.  We  have  tried  in  what 
follows  to  introduce  some  order  in  an  area  of  great  com¬ 
plexity,  but  we  have  not  attempted  any  rigid  axiomatic 
approach.  There  is  conceivably  some  purpose  in  doing  this 
in  fields  which  have  passed  their  heyday.  In  a  living 
field,  the  aim  of  theory  is  to  guide  and  stimulate,  not 
to  stultify  and  enchain. 

_ GENERATION  AND  REGENERATION 

Mathematics  is  one  of  the  tools  used  by  scientists  to 
simplify  and  unify  their  domains.  In  return,  mathematics 
is  continually  revitalized  by  the  new  problems  arising  from 
new  scientific  discoveries,  e.g.,  quantum  mechanics, 
relativity  theory,  digital  computers.  Without  this 
continuing  stimulus,  mathematics  withers  and  atrophies. 

It  is  not  a  self-sustaining  field,  although  it  can  exist 
in  embalmed  and  mummified  form. 


5.  HIERARCHIES  OF  PROCESSES 

Before  turning  to  a  classification  of  problems,  and 
ipso  facto,  of  solutions,  let  us  point  out  that  the  idea 
we  Shall  employ — that  of  a  stratification  of  elements,  a 
fomatlon  of  hierarchies — is  a  standard  one  in  mathematics. 

Vie  meet  it  in  algebra  and  in  recursive  function  theory. 
Indeed,  a  simple  hierarchic  argument  shows  the  existence  of 
transcendental  numbers.  In  connection  with  the  theory  of 
sets,  it  can  be  used  to  establish  the  existence  of 
continuous  functions  without  derivatives  at  points,  and 
so  on. 

The  Llouville  theory  of  elementary  functions  is  based 
upoh  a  sequential  concept  of  operations.  Finally,  the 
theory  of  tjTes  was  introduced  by  Russell  (see  the 
excellent  expository  article  by  Quine  [5])>  to  avoid  the 
kind  of  ambiguity  that  easily  results  from  loose  expression. 
We  see  similar  meaninglessness  today  in  discussions  of 
"thinking  machines.'' 

6.  SYSTETvlS  AND  PR0CESSE3 

V/e  propose  to  categorize  problems  by  associating  them 
with  various  types  of  "systems"  or  "processes,"  terms  whose 
significance  will  become  clearer  in  context. 

To  describe  a  system  we  require  some  further  concepts: 
state,  cause  and  effect,  and  criterion,  this  latter  only  if 
we  are  talking  about  control  processes.  By  the  state  of 
the  system,  we  mean  the  parameters  and  functions  which 
describe  it.  Thus,  by  means  of  simple  concepts,  a  stone 
thrown  into  the  air  may  be  described  at  any  time  by  its 
position  and  velocity.  A  more  sophisticated  and  realistic 
theory  will  require  further  data.  Hence,  and  a  most 
important  point,  the  state  of  a  system  is  a  relative 
concept — ^that  is,  relative  to  the  mathematical  and  physical 
theories  we  are  employing  at  the  moment. 


By  cause  and  effect,  we  mean  the  change  In  the  state 
of  a  system  resulting  fi^om  the  application  of  a  transfor¬ 
mation.  In  the  case  of  a  control  process,  a  decision  is 
equivalent  to  a  transformation,  and  we  suppose  that  we 
know  the  new  state  resulting  from  the  decision. 

We  wish  to  construct  a  hierarchy  of  processes  and  to 
indicate  which  ones  we  can  haindle  in  one  v;ay  or  another 
at  the  present  time,  and  which  ones  are  quite  remote.  In 
this  way,  we  wish  to  introduce  some  precision  into  the 
concepts  of  "intelligence"  and  "thinking." 

7.  INTELLIGENCE  AND  DECISION-MAKING 

V/e  shall  define  intelligence  as  the  ability  to  make 
decisions.  It  is  thus  easy  to  define  levels  of  intelli¬ 
gence  in  terns  of  levels  of  decision  processes.  Tnls 
interpretation  of  intelligence  is  reasonably  consistent 
with  that  given  by  psychologists,  namely,  adaptation  to 
one ‘ s  environment . 

We  shall,  for  want  of  better,  confine  ourselves  to 
decision  processes  involving  n'umbers.  The  real  diffi¬ 
culties  in  the  applications  of  the  quantitative  methods  of 
mathematics  arise  when  we  attempt  to  associate  numbers  with 
processes  vrhlch  do  not  intrinsically  possess  numerical 
utilities.  V/e  return  to  this  point  below. 

It  is  Important  to  distinguish  clearly  between  level 
of  computational  or  analytic  complexity  and  conceptual  or 
hierarchic  level.  There  is  conceptually  no  difficulty  in 
evaluating  9  raised  to  the  99th  power,  but  it  would  be 
difficult  to  do  this  quickly,  and  at  the  moment,  we  know 
no  simple  way  of  predicting  the  723rd  digit.  Similarly, 
there  is  no  conceptual  difficulty  in  partial  differential 
equations,  although  they  may  present  severe  analytic  and 
computational  difficulties.  Frequently,  the  only  way  to 
overcome  some  of  these  obstacles  is  to  use  a  higher 
conceptual  level. 


Thus,  for  example,  to  evaluate  an  Integral  such  as 


requires  real  ingenuity,  but  if  we  use  the  sophisticated 
tool  of  the  theory  of  functions  of  a  complex  variable,  it 
becomes  a  trivial  exercise.  Similarly,  the  "elementary" 
proof  of  the  prime  number  theorem,  i.e.,  without  the  help 
of  complex  variable  theory,  is  quite  difficult,  vSiereas 
the  usual  proof  is  fairly  straightfoirward* 

In  view  of  all  of  this,  we  may  cite  the  theory  of 
quasilinearization,  where  decision  processes  are  artifi¬ 
cially  introduced  to  provide  more  efficient  types  of 
analytic  approximation  and  computational  solution.  (See 
[6],  [7],  [8],  and  [lO].  See  also  [9]  where  a  two— person 
process  is  introduced  to  simplify  the  analysis.) 

Nonetheless,  there  is  value  in  constructing  a 
hierarchy  of  processes,  following  very  much  the  lines  of 
the  Llouvllle  and  Russell  theories  mentioned  above. 


At  the  simplest  level,  we  have  descriptive  processes, 
characterized  by  the  iteration  of  a  fixed  transformation 


(7.2) 


=  T(p^), 


Po  =  c. 


There  are  in  many  cases  formidable  analytic  difficulties 
in  the  path  of  carrying  out  these  instructions,  but  there 
is  no  conceptual  difficulty.  Thus,  as  mentioned  above,  the 
calculation  of  9^^  requires  a  good  deal  of  effort  if  vre 
employ  only  simple  arithmetic .  Presumably,  with  a 
sufficient  hnov^ledge  of  number  theory  we  can  obtain  more 
efficient  techniques.  We  can  thus  regard  one  of  the  princi¬ 
pal  objectives  of  mathematical  sophistication  as  the  over¬ 
coming  of  arithmetic.  Anytime  we  are  reduced  to  brute 
calculation,  we  have  made  an  admission  of  weakness. 


Descriptive  processes  of  the  foregoing  tj-pe  will  be 
called  zero-level  processes. 

8.  DIRST-LEVEL  PROCESSES 

The  first  significcint  step  in  the  direction  of 
intelligent  machines  was  made  in  the  application  of  the 
concept  of  feedback  control.  Although  an  ancient  idea, 
it  was  first  fully  brought  into  prominence  by  the 
Industrial  Revolution.  The  many  intriguing  mathematical 
aspects  of  this  concept  v;ere  early  recognized  by  Maxwell 
[llj  and  Vishnegradsky  [123;  see  [13]  for  a  reprint  of 
Maxwell's  paper  and  other  fundamental  papers  in  control 
theory. 

Referring  to  the  analytic  representation  of  (7.2), 
we  now  assume  that  at  each  stage  of  the  process  there 
exists  a  set  of  available  transformations,  T(p,q), 
where  q  designates  the  individual  transformation, 
"Feedback"  means  that  the  transformation  which  is  applied 
depends  in  some  fashion  upon  the  current  state  of  the 
system,  i.e.,  q  =  q{p). 

We  can  thus  write  as  the  analytic  equivalent  of  a 
process  of  the  first  level, 

(8-^)  Pn+1  ■ 

In  many  cases,  this  choice  of  a  transformation  (or 
"policy")  is  determined  by  an  optimization  criterion.  In 
most  cases,  we  can  pretend  that  such  an  optimization 
process  exists  and  interpret  the  actual  policy  in  these 
terms.  This  is  the  standard  "as  if"  technique  of  mathe¬ 
matical  physics  exemplified  by  Fermat's  Principle, 
Hamilton's  Principle,  etc. 

Similarly,  in  the  study  of  biological  systems  we  may 
pretend  that  organisms  behave  as  if  they  were  trying  to 
maximize  their  probability  of  survival. 


9 .  DISCUSSION 

If  we  consider  only  deterministic  processes,  we  find 
that  first— level  processes  collapse  to  zero— level  processes, 
since 


(5-1)  Pn+1  =  T(p^,q{p^))  = 

again  a  transformation  of  fixed  type.  Conversely,  we  can 
in  many  cases  gain  considerable  advantage  from  writing 
(7.2)  in  the  form  of  (9-l)  above;  see  [6],  [?]»  and  the 
discussion  in  Sec.  7- 

This  reduction  in  hierarchic  ranlc  is  due  to  the 
duality  between  point  and  line  formulation  of  Euclidean 
geometry,  an  equivalence  v/hich  masks  the  nev;  Idea  of  feed¬ 
back  control;  see  [l]  for  a  further  discussion. 

Only  v;hen  we  turn  to  stochastic  decision  processes  do 
v;e  see  the  essential  difference  between  level  zerc  and 
level  one,  and  recognize  the  true  meaning  of  "feedback." 
This  meaning  was  obscured  for  many  years. 

10.  INSTINCT 

Much  of  the  classical  argument  concerning  the 
difference  between  "Instinct"  and  "Intelligence"  can  be 
resolved  by  definition,  using  the  foregoing  concepts,  \vhat 
is  genei’ally  called  instinct  can  be  considered  to  be 
intelligence  of  level  one,  as  defined  above. 

Similarly,  a  number  of  choice  philosophical  and 
theological  conundrums  such  as  "Free  V/111"  versus 
"Predestination"  can  be  seen  to  be  unanswerable  because 
ill— defined  and  ill— posed  questions  have  been  asked. 
Naturally,  definitions  never  completely  resolve  fundamental 
questions,  but  a  formulation  of  the  type  we  are  presenting 
enables  us  to  pinpoint  basic  Issues  with  more  accuracy. 

The  problem  of  instinct  is  clearly  allied  Vfith  that 
of  information  pattern  and  discrimination,  l^/hen  the 


organism  cannot  distinguish  between  real  and  spurious 
stimuli,  we  see  such  effects  as  the  mass  suicide  of  the 
lemmings,  and  allergic  and  neurotic  behavior. 

11 .  SECON]>-LE\rEL  PROCESSES 

We  have  defined  zero— level  processes,  and  then  first- 
level  processes  as  those  requiring  a  decision  as  to  the 
operation  of  the  zero— level  process.  The  stage  has  then 
been  set  for  an  inductive  or  recursive  definition  of 
higher— level  processes. 

A  second— level  process  will  be  defined  as  one 
requiring  a  decision  concerning  the  operation  of  a  first- 
level  process.  We  encounter  processes  of  this  type  in 
connection  with  simple  learning  and  in  adaptive  control 
theory. 

Consider  "a  stochastic  control  process  of  the  following 
type.  At  each  stage  there  is  a  random  effect  which  tends 
to  disturb  the  system.  To  correct  for  this  deviation,  we 
use  feedback  control.  Suppose  that  the  random  effect  is 
represented  by  a  random  variable  r  which  has  the  simple 
distribution 


r  =  +  1,  with  probability  p, 
r  =  —  1,  vrith  probability  1  —  p. 

If  p  is  known,  we  have  a  process  of  level  one.  If  p 
is  fixed,  but  uhlcnown,  we  have  a  process  of  level  two. 

Here,  on  the  basis  of  the  history  of  the  process  over  time, 
we  maice  successively  better  estimates  of  p. 

A  detailed  discussion  of  processes  of  this  nature  will 
be  found  in  [l]. 

12.  INFINITE  HIEHARCHY  OF  PROCESSES 

If  v;e  use  the  preceding  formulation,  it  is  now  easy  to 
construct  a  hierarchy  of  processes,  each  requiring  a  more 
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sophisticated  adaptation.  Consider  the  case  wtiere  p, 
the  unknown  probability.  Is  known  to  have  a  distribution 
runctlon  dG(p,a).  Here  a  Is  an  unknown  parameter, 
which  may  Itself  (to  Introduce  a  higher  level  process)  be 
considered  to  have  a  distribution  function  dH(a,b), 
xdilch  depends  upon  an  unknown  parameter  b,  and  so  on. 

To  see  how  situations  of  this  type  arise  In  practice, 
suppose  that  we  are  picking  coins  of  the  same  kind  from  a 
barrel,  vrinere  each  coin  In  the  barrel  has  the  same  proba¬ 
bility  of  coming  up  heads.  Nov/  suppose  that  there  Is  a 
room  full  of  barrels  of  coins,  each  barrel  having  its  ovm 
probability  distribution,  and  a  building  full  of  barrel- 
packed  rooms,  a  warehouse  center  full  of  such  buildings, 
and  so  on. 

We  see  then  that  we  can  construct  a  hierarchy  of 
decision  processes  and  thus  a  hierarchy  of  levels  of 
Intelligence.  Simultaneously,  using  standard  procedures, 
we  can  construct  decision  processes  not  belonging  to  any 
member  in  this  hierarchy.  Where,  for  example,  do  we  put 
the  problem  of  detej-Tnlnlng  at  vdnat  level  a  particular 
process  belongs? 

13.  CREATIVITY 

It  appears  that  If  we  vri.sh  to  talk  about  creativity 
In  some  reasonable  fashion,  we  must  Introduce  similar 
concepts  to  those  mentioned  above.  We  want  to  distinguish 
between  innovation  within  the  framework  of  a  particular 
decision  level  and  Innovation  outside  of  this  framework. 

It  Is  not  difficult  to  see  that  we  possess  very  feeble 
means.  If  any,  for  transcending  hierarchies  and  that  to 
contemplate  this  type  of  activity  by  means  of  digital 
computers  Is  rather  absurd.  The  use  of  raridom  stimuli  to 
simulate  human  thirlclng  Is  too  reminiscent  of  Gulliver's 
experiences  In  the  laboratories  of  Laputa  to  be  taken 
seriously. 
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14.  INCONSISTENT  lOGICS  AND  RA.MM  AXIOM  SYSTMS 

ClassicaJ.  logical  systems  havei  insisted  upon  the 

development  of  approaches  to  problem  solving  which  yield 
correct  answers  vrithout  fail.  Tiiisj  is  a  luxury  to  which 
one  cannot  aspire  in  more  complex  Situations,  principally 
because  we  do  not  understand  enough  about  them  to  formulate 
the  problems  in  completely  meaningful  terms.  V/e  conse¬ 
quently  develop  a  set  of  local  logics  each  applicable  to  a 
neighborhood  of  processes.  These  peed  not  be  consistent 
vrtien  they  overlap,  and  in  general  will  not  be.  There  is 
thus  nothing  more  illogical  than  trying  to  carry  argimients 
1  ^  their  "logical"  conclusion  in  the  political,  social  and 
economic  spheres.  It  is  a  denial  of  the  history  of 
science  to  look  for  "unified"  theories  of  complex  phenomena. 
If  it  cannot  be  done  in  the  physical  sciences,  hov;  then  in 
the  more  difficult  fields  of  human  behavior? 

This  does  not  mean  that  we  allov?  irrationality  freely. 
On  the  contrary,  the  whole  endeavor  of  the  intellectual 
approach,  the  so-called  scientific  method,  is  to  reduce 
the  regions  of  irrationality.  If  we  are  Interested  in 
actually  solving  problems  by  means  of  digital  computers, 
it  may  be  more  efficient  to  lase  a  nimiber  of  different 
approximate  logics,  with  constant  control  and  comparison. 

If  we  want  to  minimize  the  time  required  to  resolve  a 
question,  we  may  be  able  to  accomplish  this  vrlth  a  set  of 
approximate  policies,  tried  one  alter  the  other,  rather 
than  with  one  ponderous  technique  which  has  complete 
certitude . 

Closely  allied  ^fith  this  is  the  investigation  of 
situations  in  which  a  variety  of  consistent  logicfil  systems 
are  used  in  a  random  fashion.  Sijnulation  processes  of  this 
nature  could  produce  some  interesting  results. 

15.  MACHINTjIS  that  play  chess  IN  THE  NIGHT 

The  best  chess— playing  machines  to  date  were  built  in 
the  nineteenth  century.  They  used  the  very  simple  idea  of 


-12- 


having  a  human,  hidden  inside.  With  the  introduction  of 
large-scale  computers,  the  idea  of  constructing  electronic 
chess  players  was  pursued  again  by  a  number  of  enthusiasts. 
This  is  a  very  intriguing  idea,  but — like  many  ideas  of 
this  nature — one  that  can  lead  to  a  substar'xtial  waste  of 
time  and  effort  if  no  adequate  preliminary  evaluation  is 
made . 

It  is  rather  unfortunate  that  digital  computers,  which 
were  designed  by  such  mathematical  pioneers  as  V/iener  and 
von  Neumann  to  resolve  a  number  of  significant  scientific 
questions,  have  to  such  a  great  extent  been  relegated  to 
accounting  and  bookkeeping  and  to  the  routine  solution  of 
routine  engineering  problems.  It  is  even  more  unfortunate 
that  the  small  amount  of  time  left  over  from  these  profit¬ 
able  (?)  activities  has  been  devoted  to  the  calculation  of 
prime  numbers  and  to  the  development  of  chess— playing 
programs . 

Way  is  chess  a  poor  game  to  analyze  by  computer? 

In  the  first  place,  it  is  not  a  game,  such  as  tic— tac— 
toe,  which  can  be  resolved  by  merely  an  enumei^ation  of 
possible  combinations  of  moves.  Even  the  game  of  checkers, 
far  more  limited  in  strategic  and  tactical  concepts,  cannot 
be  played  on  a  master  level  by  computer,  although  sophis¬ 
ticated  enumeratlve  and  feedback  techniques  can  be  used  to 
program  a  computer  to  play  a  strong  game. 

Chess  requires  a  deeper  vinderstanding — vrfiich  vie  do  not 
possess  at  the  present  time.  Essentially  the  difficulty  is 
that  tactics  and  strategy  intertwine  In  complex  ways. 

Although  the  openings  are  fairly  standard,  the  middle  game 
is  an  uncharted  Jungle. 

In  natural  situations  in  which  we  do  not  possess  precise 
optimal  policies,  we  must  employ  approximate  techniques. 

This  is  where  the  imnatural  character  of  chess  as  a  contrived 
process  manifests  Itself.  Chess  is  an  Inhei’ently  unstable 


game.  By  this  we  mean  that  a  small  difference  in  the 
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posltlon  of  the  pieces  and  pawns  can  make  the  difference 
betv^een  winning  and  losing.  Even  the  number  of  pieces 
and  pawns  Is  frequently  of  no  significance.  Thus  approx¬ 
imation  techniques  are  of  questionable  value. 

The  concept  of  checkmate  is  an  interesting  but  higbily 
artificial  one.  Just  the  opposite  is  true  of  physical 
prrcesses  v/here  there  is  necessarily  stability,  a  funda¬ 
mental  observation  of  Hadamard,  and  where  approximate 
policies  work  extremely  v:ell — fortiinately  for  physicists 
and  engineers. 

Let  us  note,  hov/ever,  that  this  opinion  of  the 
feasibility  of  constructing  chess— playing  programs  is  not 
shared  by  the  world  champion,  Botwinnik,  who  is  actually 
engaged  in  investigations  of  this  type.  On  the  other 
hand,  another  great  master,  Edward  Lasker,  shares  the 
opinions  expressed  above. 

16.  DYT-JAIilC  PROGRAI-PilNG 

As  v.'e  have  already  mentioned,  the  hierarchy  of 
decision  processes  we  constructed  was  considerably 
Influenced  by  the  a  pi'iori  knowledge  of  a  mathematical 
theorj'-  that  could  formulate  the  resultant  mathematical 
problems  in  precise  analytic  terms,  cind,  occasionally,  in 
happy  circumstances,  resolve  them  analytically  and 
computationally . 

The  basic  idea  is  that  of  a  policy  which  is  a  sequence 
of  decisions,  each  of  which  is  based  upon  current  informa¬ 
tion.  This  information  pattern,  or  state  variable,  is  the 
clue  to  the  complexity  of  the  decision  process.  If  the 
state  variable  is  an  n— dimensional  vector,  or  even  a 
function,  we  have  a  process  of  level  one.  If  it  is  a 
distribution  function  for  a  vector  in  n— dimensional  space, 
we  have  a  process  of  level  two.  If  it  is  a  distribution 
function  for  parameters  wiaich  determine  a  distribution 
function,  we  have  a  process  of  level  three  and  so  on. 
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We  see  then  that  we  could  employ  the  Llouvllle  type 
of  classification  to  construct  a  hierarchy,  without  any 
reference  to  the  original  decision  processes.  This  is 
convenient,  but  an  identification  must  still  be  made  at 
some  point. 

Detailed  discussion  of  deterministic,  stochastic, 
and  adaptive  processes  will  be  found  In  [l]. 

17.  MATHEMATICAL  EXPERIME^^^TATION 

Since  we  know  so  little  about  self— organizing  systems, 
we  must  carry  out  a  large  number  of  trial  Investigations. 

In  other  words,  we  must  engage  in  mathematical  experi— 
m.entatlon.  One  of  the  most  useful  tools  the  mathematician 
possesses  is  the  digltaJ.  computer. 

At  the  present  time,  a  good  deal  of  work  is  going  on 
in  connection  with  learning  and  adaptive  processes,  but 
not  nearly  enough  compared  to  the  magnitude  of  the  diffi¬ 
culties  and  the  great  variety  of  different  problems 
confronting  us. 

This  does  not  mean  that  we  should  search  randomly. 

It  does  mean,  that  we  should  investigate  the  behavior  of 
specific  mathematical  models,  the  efficiency  of  various 
types  of  approximation,  particularly  approximation  in 
policy  space,  and  so  on. 

In  particular,  we  must  find  simple  approximate  tech¬ 
niques  for  reducing  hierarchic  level  and  for  using  as  much 
of  a  quantitative  method  as  possible  in  the  study  of 
qualitative  phenomena.  If  we  are  to  avoid  the  morass  of 
metaphysics,  we  must  reduce  as  many  concepts  as  possible 
to  numerical  terms.  On  the  other  hand,  we  must  face  the 
fact  that  the  most  important  aspects  of  himian  life  are 
intrinsically  nonnumerical.  Any  att(2mpt  to  ignore  this  is 
highly  vinscientific .  In  the  true  intellectual  approach, 
one  accepts  this  fact  and  copes  with  it. 
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